System and method for real-time ultrasound guided prostate needle biopsies using a compliant robotic arm

ABSTRACT

Systems and methods are provided for utilizing an MRI image and real-time an ultrasound images to guide and/or restrict the movement of an ultrasound probe in position for collecting a biopsy core. A real-time ultrasound image is acquired and fused with pre-operative imaging modalities, such as an MRI image, to provide a three-dimensional model of the prostate. A multi-link robotic arm is provided with an end-effector and an ultrasound probe mounted thereto. Sensor information is used to track the ultrasound probe position with respect to the 3D model. The robotic arm allows for the implementation of a virtual remote center of motion (VRCM) about the transrectal probe tip, an adjustable compliant mode for the physician triggered movement of probe, a restrictive trajectory of joints of the robotic arm and active locking for stationary imaging of the prostate.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present patent document claims the benefit of U.S. PatentApplication No. 62/013,850, filed on Jun. 18, 2014, which is herebyincorporated by reference.

BACKGROUND

It is common to diagnose prostate cancer using transrectalultrasonography (TRUS) guided needle biopsy procedures. A TRUS guidedneedle biopsy is often prescribed as a result of an elevatedprostate-specific antigen (PSA) level or upon detecting a palpablenodule during a digital rectal exam. Introduction of ultrasound (US)image-guided prostate biopsies substantially increases the accuracy of aprostate biopsy over performing a blind biopsy, thus TRUS guidance hasbecome a universally accepted method for prostate biopsy. While TRUSguided prostate biopsies are a clinically accepted method, theprocedures retain a low sensitivity of approximately 60% and positivepredictive value of only approximately 25%. Consequently, repeatbiopsies are often required. For example, in over 20% of the cancerstudies, more than one biopsy session is required for a physician toreach a diagnosis decision.

Three-dimensional visualization applications, such as magnetic resonanceimaging (MRI), can clearly depict the prostate gland and prostatesubstructures, including the central, transitional, and peripheralzones. For example, T2-weighted images can depict nodules in theperipheral zone. Localizing a tumor foci and the peripheral zone withMRI before a prostate biopsy may increase the overall cancer detectionrate and the biopsy yield. Additionally, functional information acquiredfrom various MRI techniques, such as diffusion weighted imaging (DWI),dynamic contrast-enhanced (DCE) imaging or chemical shift imaging (CSI),may be used to further characterize the prostatic tumor tissue. Usingthis functional information during the ultrasound guided biopsy maysubstantially improve the sensitivity of the biopsy procedure.

For example, endorectal MRI images and findings for suspected tumor focimay be used to manually guide the placement of needles during TRUSguided biopsy. By localizing biopsy to suspected tumor lesions and othertargets identified on the endorectal MRI image, the physician mayvisually correlate the locations in the endorectal MRI image withultrasound images during a subsequent TRUS guided biopsy, increasing theaccuracy of the TRUS guided biopsy to approximately 67%, as demonstratedin a study of 33 patients. However, correlating locations using an MRIimage requires a tedious visual inspection.

Robotic arms have been developed for handling ultrasound probes. Forexample, the robotic arms may alleviate physician fatigue duringscanning procedures. In a robotic system, a physician's grasp and inputto the ultrasound probe may be detected, such as by a 6-axis forcesensor, and multiple detected parameters may be used for compliancecontrol and self-weight compensation of the ultrasound probe as theprobe is moved and adjusted by the physician.

SUMMARY

The present embodiments relate to real-time ultrasound guided prostateneedle biopsies using a compliant robotic arm. By way of introduction,one or more of the present embodiments include systems and methods forutilizing an MRI image and real-time ultrasound images to guide and/orrestrict the movement of an ultrasound probe in position for collectinga biopsy core. A real-time ultrasound image is acquired and fused with apre-operative imaging modality, such as an MRI image, to provide athree-dimensional model of the prostate. A multi-link robotic arm isprovided with an end-effector and an ultrasound probe mounted thereto.Sensor information is used to track the ultrasound probe position withrespect to the 3D model. The robotic arm allows for the implementationof a virtual remote center of motion (VRCM) about the transrectal probetip, an adjustable compliant mode for the physician triggered movementof probe, a restrictive trajectory of joints of the robotic arm andactive locking for stationary imaging of the prostate.

In a first aspect, a method for ultrasound guided prostate needlebiopsies using a compliant user-driven robotic arm is provided. Athree-dimensional image of a prostate is received and athree-dimensional model of the prostate is generated using thethree-dimensional image. Using the compliant user-driven robotic arm, anultrasound probe is guided into a position for a three-dimensionalultrasound sweep of the prostate and the three-dimensional ultrasoundsweep of the prostate is performed using the ultrasound probe in theposition. The three-dimensional model of the prostate is updated usingimage data from the three-dimensional ultrasound sweep. Based on theupdated three-dimensional model of the prostate, a three-dimensionallocation is identified in the prostate for collecting a biopsy core,where a location from the three-dimensional diagnostic image is mappedto corresponding location from the image data from the three-dimensionalultrasound sweep. The ultrasound probe is guided into a position tocollect the biopsy core with the robotic arm by restricting thetrajectory of the robotic arm based on the identified three-dimensionallocation in the prostate and displaying the identified location.

In a second aspect, a system for ultrasound guided prostate needlebiopsies using a robotic arm is provided. A processor is configured toreceive a three-dimensional image of a prostate, wherein the processorgenerates a three-dimensional model of the prostate based on thereceived image. A multi-link robotic arm includes a plurality of joints,an end-effector attached to one of the plurality of joints and anultrasound probe attached to the end-effector. The ultrasound probe isoperably connected to the processor in order to capture an ultrasoundimage, wherein the processor updates the three-dimensional model of thepatient based on the captured ultrasound image. A position sensor isprovided for at least one of the plurality of joints. The positionsensor is operably connected to the processor to map the position of theultrasound probe with the updated three-dimensional model. A forcesensor is operably connected to the processor. The force sensor isconfigured to detect input forces to the end-effector. The presentinvention is defined by the following claims, and nothing in thissection should be taken as a limitation on those claims. Further aspectsand advantages of the invention are discussed below in conjunction withthe preferred embodiments and may be later claimed independently or incombination.

BRIEF DESCRIPTION OF THE DRAWINGS

The components and the figures are not necessarily to scale, emphasisinstead being placed upon illustrating the principles of theembodiments. Moreover, in the figures, like reference numerals designatecorresponding parts throughout the different views.

FIG. 1 illustrates an embodiment of a system for ultrasound guidedprostate needle biopsies using a robotic arm.

FIGS. 2A-C depict examples of prostate deformation caused by anultrasound probe.

FIGS. 3A-B depict examples of a prostate segmented in an MRI image andan ultrasound image.

FIG. 4 illustrates a flowchart of an embodiment of a method forultrasound guided prostate needle biopsies using a robotic arm.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Current prostate biopsy procedures are performed systematically underultrasound imaging guidance and often lack sensitivity in detectingmalignant lesions. The present embodiments may increase the precision ofprostate biopsy procedures used to detect cancer. To enhancesensitivity, a high quality three-dimensional scan, such as preoperativeMRI data, is used during an ultrasound guided biopsy procedure.Information from the preoperative MRI data is used to determine anoptimal placement of the biopsy needle cores in order to take accuratepathological samples, thus increasing the detection yields of thebiopsy. For example, a real-time ultrasound image is fused with an MRIimage to better target lesions within the prostate that are identifiedon the MRI image. An active compliant robotic arm is then used to trackthe position of the ultrasound probe attached to the end effector of therobotic arm. The user controls motion of the ultrasound probe by holdingthe probe handles, and the system detects the input force provided bythe user measured by a decoder on the handle. The system also tracks theposition of the ultrasound probe and the joints of the robotic arm inthree-dimensions. The information received from of the ultrasound probeand the robotic arm allow for an active compliant and restrictive motionschemas for the robotic arm implemented within the control loop of therobot using the kinematics of the system. Further, a locking mechanismis provided that prevents all movement of the ultrasound probe and therobotic arm.

The disclosed embodiments may address challenges in fusion-basedprostate biopsy by minimizing scanning variability, by addressing theeffects of prostate deformation, by increasing ease of use and byincreasing the precision in transferring MRI localized targets onto areal-time ultrasound sequence for guidance of the ultrasound probe. Indoing so, a control loop allows the robotic arm to passively or activelymanipulate the ultrasound probe held by the user using the real-timeultrasound images fused with pre-operative imaging modalities.

As such, one or more of the present embodiments provide an activecompliant schema and restrictive motion, such as the trajectory of therobotic arm. For example, the trajectory of the robotic arm may berestricted by a virtual remote center of motion (VRCM or RCM) about theultrasound probe tip. Further, automatic probe movement and scanning maybe provided via the remote center of motion, or compliant movement thatonly allows the user to move the ultrasound probe in a specified manner(e.g., to maintain coupling pressure on the rectal wall). In anotherexample, image analysis and feature tracking automatically adjusts thecoupled pressure of the probe on the rectal wall and/or moves the probeto target an image-based location in the prostate. In another example,adjustable compliance of the robotic arm is provided. For example,compliance during any mode of operation, such as a “complaint” or“automatic” mode, restricts movement of the robotic arm based on aspecific trajectory using a three-dimensional model of the prostateand/or a virtual remote center of motion of the probe.

FIG. 1 illustrates an embodiment of a system for ultrasound guidedprostate needle biopsies using a robotic arm. The system 100 includes amulti-link robotic arm 101 and a workstation 109. The multi-link roboticarm 101 includes a plurality of joints 103, an end-effector 105 and anultrasound probe 107. The robotic arm 101 is attached to an ultrasoundmachine, such as the workstation 109, or is freestanding, allowing therobotic arm 101 to be freely positioned by the user for better patientaccess. The end-effector 105 is attached to one of the plurality ofjoints 103, and the ultrasound probe 107 is attached to theend-effector. The multi-link robotic arm 101 also includes forcesensors, position encoders, optical decoders and/or other sensors (notdepicted in FIG. 1) provided at one or more of the plurality of joints103. The workstation 109 includes a computer 111, a user interface 113and a display 115. Additional, different, or fewer components may beprovided.

The computer 111 is a computer platform having hardware such as one ormore central processing units (CPU), a system memory, a random accessmemory (RAM), a graphics processor unit (GPU) and input/output (I/O)interface(s). Additional, different or fewer components may be provided.For example, the computer 111 may be implemented on one or more servers.In another example, workstation 109 includes a controller forcontrolling the robotic arm in addition to computer 111. In anotherexample, the computer 111 is separate from or not part of theworkstation 109.

The computer 111, or a processor therein, is configured to receive athree-dimensional representation of a prostate, such as preoperative ordiagnostic MRI image or data. As used herein, the MRI image may be datafrom a scan prior to display, so may include a three-dimensionalrepresentation. Based on the three-dimensional image, the computer 111is further configured to generate a three-dimensional model of theprostate. The three-dimensional model may utilize biomechanicalproperties of the prostate determined (i.e., Young's modulus and thePoisson's ratio) using nominal values reported from literature, or frompersonalized values extracted from elastography scans of the patient.Using the three-dimensional image and/or the generated three-dimensionalmodel of the prostate, one or more biopsy cores are identified manuallyor automatically. The computer 111, or a processor therein, is furtherconfigured to receive one or more real-time ultrasound images capturedby the ultrasound probe 107 operably connected therewith. Using thereal-time ultrasound image, the computer 111, or a processor therein, isconfigured to generate an updated three-dimensional model. For example,the ultrasound probe 107 captures an ultrasound sweep of the prostate,and the three-dimensional model is updated to account for prostatedeformation caused by the ultrasound probe. Further, feature locationsidentified using the preoperative MRI are updated in the updatedthree-dimensional model to account for the effects of the prostatedeformation. In this example, one or more biopsy cores are identifiedusing the updated three-dimensional model of the prostate.

The computer 111, or a processor therein, is configured to track thelocation and movement of the joints of the multi-line robotic arm 101,the end-effector 105 and/or the ultrasound probe 107. For example, aposition sensor (not depicted in FIG. 1), such as an optical decoder ora position encoder, is attached to each of the plurality of joints 103and is operably connected to the computer 111. The computer 111 isconfigured to map the position of the ultrasound probe, such as withrespect to the updated three-dimensional model of the prostate. Usingsensors or calibration, the position of the robotic arm 101 in thecoordinate space of the MR or other preoperative system is known.Further, the computer 111, or a processor therein, is configured toreceive user inputs at the end-effector 105 and/or the ultrasound probe107. For example, a force sensor (not depicted in FIG. 1) is provided inclose proximity to the joint 103 where the handle of the ultrasoundprobe 107 is attached to the end effector 105. In another example, theforce sensor is attached to the handle of the ultrasound probe.

As discussed above, the position encoders at the joints 103 detect thekinematics of the robotic arm, allowing for real-time sensing of theultrasound probe 107 location at all times during the procedure. Anultrasound calibration process is performed, with or without a phantom,to determine a transformation used to map ultrasound image points to afixed coordinate system based on the location of the joints 103 of therobotic arm and the ultrasound probe 107 location. The real-time sensingof the joint 103 and the ultrasound probe 107 locations facilitateactive or passive control over the robotic arm 101, and ultimately, theultrasound probe 107. For example, the end-effector 105 or theultrasound probe 107 includes a handle allowing the user to freely movethe probe as required within the space allowable by the robotic arm'skinematics.

The robotic arm 101 is configured, via the computer 111, to operate in aplurality of operating modes to control the movement of the joints 103and the ultrasound probe 107. For example, the robotic arm 101 operatesin freestyle mode, active compliant mode, locking mode, RCM mode,automatic mode or a combination thereof. In freestyle mode, motion ofthe robotic arm is triggered by a user input and the motion is onlylimited by the structure of the joints (e.g., the mechanical design ofthe robotic arm).

In active compliant mode, motion of the robotic arm is triggered by auser input sensed by the force sensor and the robotic arm facilitates“weightless” movement of the end-effector. Further, the active compliantmode facilitates constraints on movement, such as restricting movementof the robotic arm to a specific trajectory or with respect to aparticular location, such as a trajectory to perform a task or to avoidcollision with another object in the room. For example, the control loopfor the robotic arm senses a user input and assists the user in movingthe robotic arm in the direction of the input (e.g., passively guidingmovement of the robotic arm). As soon as the user input ceases, therobotic arm maintains its current position until another user inputinitiated.

In a locking mode, the joints 103 of the robotic arm 101 activelymaintain their positions (i.e., restrict all motion of the robotic arm).In an example, the locking mode resists motion up to a certain thresholdof exerted force on the end-effector 105, the handle of the ultrasoundprobe 107 and/or the joints 103.

In an RCM mode, motion of the robotic arm is restricted to movementabout a fixed point, such as the distal end of the ultrasound probe. Forexample, the user manually performs a three-dimensional ultrasound sweepof a prostate by moving the ultrasound probe about a fixed point. Inthis example, the robotic arm restricts movement of the tip of theultrasound probe, by keeping the tip of the probe in a fixed location,and allows the user to rotate the ultrasound probe about the tip toperform the ultrasound sweep. Thus, the robotic arm is passively guidesthe ultrasound probe by restricting the user's ability to manipulate theprobe about the fixed location.

In an automatic mode, the ultrasound probe is moved automatically by therobotic arm. For example, the robotic arm automatically performs athree-dimensional ultrasound sweep of a prostate by moving theultrasound probe about a distal end of the probe while capturingultrasound images (e.g., actively guiding movement the robotic arm via aremote center of motion). The computer 111 or other processor controlsthe movement without user input other than activation. In anotherexample, the robotic arm automatically moves the ultrasound probe intoposition to collect a biopsy core. The robotic arm automatically movesthe ultrasound probe based on the detected ultrasound probe location,the detected robotic arm joint locations, the three-dimensional model ofthe prostate, the preoperative three-dimensional scan and/or the captureultrasound images.

Additional, different or fewer modes and/or constraints on movement maybe provided. For example, additional constraints may be included in anyof the above mentioned modes, such as to avoid self-collision, collisionwith ultrasound machine, patient discomfort, and other constraintsrelated to the prostate biopsy procedures. In another example, therobotic arm is restricted based on analysis of captured ultrasoundimages. The modes are user selectable. For example, the locking mode istriggered by a push button. Alternatively, the modes are automaticallyselected based on a system state or a workflow.

For example, in one or more modes discussed above, movement of theultrasound probe is guided, constrained and/or restricted to maintainpressure on the prostate. For example, the robotic arm passively oractively guides the ultrasound probe to maintain consistent couplepressure of the ultrasound probe with the rectal wall in compliant mode,RCM mode and automatic mode. An ultrasound probe requires adequatecoupling pressure to capture an ultrasound image, however, limited anduniform prostate deformation is also preferred and results in accurateultrasound images. For example, FIGS. 2A-C depict examples of prostatedeformation caused by an ultrasound probe. FIGS. 2A-B depict twopositions of freehand sweep for biopsy cores (e.g., in a freestylemode). As the ultrasound probe sweeps from the position in FIG. 2A tothe position in 2B, the prostate is deformed differently, resulting ininconsistent ultrasound images of the prostate. The deformation causedby different positioning and/or pressure may cause the lesion or regionto be biopsied to alter position with the prostate and/or relative tothe probe. Conversely, FIG. 2C depicts an ultrasound sweep (e.g, in thecomplaint, RCM or automatic mode) minimizing prostate deformationvariation between various probe positions. In addition to RCM mode, forexample, coupling pressure and prostate deformation can be controlled bymonitoring the captured ultrasound images (e.g., by monitoringde-correlation between consecutively captured ultrasound images, bymonitoring a difference in a strain on prostate tissue betweenconsecutively captured ultrasound images, by generating an imageconfidence map based on consecutively captured ultrasound images, or acombination thereof).

One or more of the above embodiments is provided by a control loop forthe robotic arm. For example, the control loop can passively guide theultrasound probe by receiving a user input and restricting thetrajectory of the robotic arm based on constraints provided in thecontrol loop. In another example, the robotic arm actively guides theultrasound probe. In these example, image based constraint drivencontrol of the robotic arm is provided by the control loop, such as bycontrolling the trajectory of the robotic arm. The image basedconstraints are derived by acquiring and fusing real-time ultrasounddata with a pre-operative data modality, and mapping image-basedconstraints derived from the fused data to the coordinate space of therobotic arm. Constraint driven control in robotics provides constraintson the movement of the robotics, such as on the range of motion of arobotic arm. For example, constraint driven control can be used to applyconstraints in order to perform specific tasks (e.g., task basedconstraints). For example, a remote center of motion is a task basedconstraint. One or more of the present embodiments implement image-basedconstraints in a control loop schema for a robotic arm.

The image-based constraints used in a control loop can be implementedusing various implementations. The following equations and othermathematical representations are provided for illustrative purposes.Other implementations, equations and mathematical representations may beused.

For example, a constrained control schema for a robotic arm can bewritten as equation 1:

$\begin{matrix}{{{\Delta\; q} = {\min\limits_{\Delta\; q}{\sum{{\mathcal{g}}_{i}\left( {{\Delta\; q},{\Delta\; x_{i}^{d}}} \right)}}}};{{s.t.\mspace{14mu}{c_{j}\left( {{\Delta\; q},{\Delta\; x_{j}}} \right)}} = 0}} & \left( {{Eq}.\mspace{14mu} 1} \right)\end{matrix}$where Δq is the incremental motion of the joints of the robotic arm(e.g., if the robotic arm has three joints, then Δq is includes threevectors representing the incremental motion of the joints). The jointvelocities of the robotic arm are linearized using the derivativeΔq/Δt={dot over (q)}, where {dot over (q)} is joint velocities.Incremental Cartesian motion of the end-effector is linearized as Δx_(i)and Δx_(j), and functions g_(i) and c_(j) relate the incrementalCartesian motion of the end-effector to the incremental joint motion(e.g., relationships between end-effector motion and joint motion).Using the minimization of equation 1, a desired trajectory of theend-effector and joints of the robot is achieved taking into account thecurrent end-effector and joint orientation and speed, and subject to oneor more constraints placed on the movement. The same notation can beextended for an image based constrained control schema.

An image based Jacobian matrix can be applied to the constraint controlschema to provide a linearized relationship between image features andthe change in position of the end-effector and joints of the roboticarm. Using the image based Jacobian matrix, constraints are derived fromimage features in the image space, such as an RCM point or an identifiedbiopsy location. Alternatively, an ultrasound probe based image Jacobianmatrix can be applied to the constraint control schema to provide alinearized relationship between Cartesian features and the change inposition of the end-effector and joints of the robotic arm. Using theultrasound probe based Jacobian matrix, Cartesian based constraints fromthe Cartesian space are derived from features in the Cartesian space,such as the patient table and other equipment in the Cartesian space.Features in the Cartesian space are identified during calibration andare used for collision avoidance constraints. In an embodiment,constraints from the image space and the Cartesian space are applied inrestricted space (i.e., areas where image based and/or Cartesian basedfeature constraints have been derived), restricting and/or guidingmotion of the end-effector and joints of the robotic arm. In theunrestricted image and Cartesian space (e.g., areas where no constraintshave been derived), movement of the robotic arm by the user isunconstrained (e.g., in a freestyle or complaint mode).

In an embodiment, applying image-based constraints, f is a set ofobserved features in the image space that are included in avectorization of an entire image, and Δf is the corresponding vector ofrate of change of image features. An image Jacobian is used to perform alinear mapping from the tangent space of x_(k) (e.g., the Cartesianposition of the end-effector, or any other arbitrary coordinate frameattached to the robot) to the tangent space of the image features (e.g.,Δf), written as equation 2:Δf=J _(I) Δx _(k)  (Eq. 2)Applying the manipulator Jacobian to equation 2 provides the linearizedrelationship between Cartesian position of the k^(th) coordinate frameto the rate of change of joints of the robotic arm, written as equations3 and 4:Δf=J _(I) J _(k) Δq  (Eq. 3)where, Δx _(k) =J _(k) Δq  (Eq. 4)

Using the image Jacobian from equations 2-4, equation 1 is rewritten, asprovided in equation 5:

$\begin{matrix}{{{\Delta\; q} = {{\min\limits_{\Delta\; q}{{{\Delta\; x_{i}^{d}} - {J_{i}\Delta\; q}}}} + {\sum\limits_{j = 1}^{N}{{{\Delta\; x_{j}^{o}} - {J_{j}\Delta\; q}}}} + {\alpha^{t}s_{1}} + {\beta^{t}s_{t}}}}{{{s.t.\mspace{14mu}{{{\Delta\; x_{k}} - {J_{t}J_{k}\Delta\; q}}}_{2}} \leq {ɛ_{t} + {s_{1}{\forall k}}}} = {1\mspace{14mu}\ldots\mspace{14mu} M}}{{{{{\Delta\; x_{t}} - {J_{t}\Delta\; q}}}_{2} \leq {ɛ_{t} + {s_{t}{\forall t}}}} = {1\mspace{14mu}\ldots\mspace{14mu} P}}} & \left. {{Eq}.\mspace{14mu} 5} \right)\end{matrix}$where J_(k), J_(t), J_(i) and J_(j) are manipulator Jacobians for framesk, t, i and j, respectively, where J_(I) is the image Jacobian. Δx_(k),Δx_(t), Δx_(i) and Δx_(j) are the incremental Cartesian positions forframes k, t, i, and j, respectively. The superscripts d and o refer tothe desired incremental position of the ultrasound probe or handle i andthe objective to be met j, respectively. In equation 5, there are P taskbased constraints, M image based constraints and N objectives. The taskbased constraints (e.g., RCM), image based constraints (e.g.,constraints based on the ultrasound image and/or the preoperative MRIimage) and the objectives (e.g., constraints based on the mechanicallimits of the robotic arm) are met in addition to facilitating thedesired movement of the ultrasound probe. Slack variables s_(I) ands_(t) may be included to relax the constraints, wherein the degree ofslackness is controlled by parameters α and β, respectively.

Previously, the aforementioned constraints were either expressed aslinear or nonlinear constraints. For example, constraint linearizationhas been accomplished by approximating a non-linear feasible data setapproximating a closed polygon with n-sides. Nonlinear constraints havebeen solved using feasible sequential quadratic programs that convertthe nonlinear constraints into a series of iterative quadratic equationswhile ensuring that the solutions remain within the feasible set. Inthese approaches, a trade-off between speed (e.g., via linear constraintmethods) and accuracy (e.g., non-linear constraints) was made.

Trade-offs between speed and accuracy may not be required where noconstraint linearization is performed. In an embodiment, a second-ordercone programming (SOCP) is solved directly. For example, an objectivecan be expressed as any convex form of L1, L2 or L2 norm squared.Because each convex form has the same minimum associated therewith, theconvex forms can be substituted for one another. The SOCP can be solvedby algorithms such as an embedded conic solver (ECOS), which usesinterior point methods for SOCP. As a result, it may be possible tosolve this problem in the rates required for robot control (e.g.,typically in less than 10 milliseconds).

In one or more embodiments, one or more of the following illustrativeconstraints are provided for ultrasound guided prostate biopsy with arobotic arm. For example, constraints are implemented using equation 5with the SOCP solved directly. Additional, different or fewerconstraints may be proved. For example, one or more constraints may beimplemented unrestricted (e.g., allowing the robotic arm to be movedfreely), as a soft constraint (e.g., resisting motion of the robotic armin a specified way) or a hard restraint (e.g., preventing all motion ofthe robotic arm in a specified way).

Compliant adjustment and locking of the robotic arm may be helpful tothe user to assist accurate movement of the ultrasound probe and tolimit fatigue associated with ultrasound procedures. In an embodiment,compliant adjustment and locking of the robotic arm is provided. Forexample, compliant control of the robotic arm is provided usingimpedance control (e.g, via the relationship between force and position)or admittance control (e.g., the inverse of impedance control) of therobotic arm. For example, using admittance control (or pseudo admittancecontrol), a lower level controller may be used, such as aproportional-integral-derivative (PID) controller that can drive therobotic arm to set an orientation and/or velocity of the robotic arm. Inthis example, a sensor is attached to the handle of the ultrasoundprobe, such as a force senor or an optical encoder capable of generatinga signal proportional to user input. Therefore, if τ is the 6D userinput from this sensor (e.g., force and torque measurements in theeCartesian coordinates), then the desired incremental position of theultrasound probe can be mapped using equation 6:

$\begin{matrix}{{\Delta\; x_{i}^{d}} = {K\;\tau}} & \left( {{Eq}.\mspace{14mu} 6} \right)\end{matrix}$where K is a 6×6 scaling matrix. In compliant mode, the robotic armreceives a user input via the sensor and assists the user in moving therobotic arm, such as allowing “weightless” motion in response to aninput. Further, for a locking mode, τ or the scaling matrix K are set tozero upon triggering the locking mode (e.g., when the system receives nouser inputs or the locking mode is initiated by the system and/or user).

Virtual remote center of motion (VRCM or RCM) is a useful constraint ofa medical robot control, such as to limit inconsistent prostatedeformation. In an embodiment, a RCM for the robotic arm is provided.For example, the system or user directly selects the RCM point based onan image feature. For instance, the user can select a feature in theultrasound image, the MRI image and/or a three-dimensional model, suchas the apex of a prostate in the ultrasound image. A Jacobian is thencomputed for the selected feature with respect to joints of the roboticarm to implement the RCM about the selected feature set as theconstraint. Therefore, if x_(p) is the 3D coordinates of the desired RCMpoint in image space, then the RCM constraint can be written as equation7:|x _(p) +Δx _(p) −J _(I) J _(p) Δq| ₂≤ϵ_(I)  (Eq. 7)Unlike earlier RCM approaches, it may not be required that the RCM pointbe known to the system in Cartesian coordinates with respect to theend-effector of the robot or the ultrasound probe. Further, as discussedabove, solving conic problems with interior point methods allowsequation 7 to be solved directly instead of solving a linearized form ora locally linear solution. This is important if the tolerance of the RCMis crucial, such as in prostate TRUS procedures. For example, if theultrasound probe tip moves too far away from the RCM point variesprostate deformation, causing a potential error in the needle guidance.

Further, it may be desirable to swiftly acquire a three dimensionalvolume of the prostate by compounding 2D US images by performing anultrasound sweep. It may also be desirable limit prostate deformationduring ultrasound sweep acquisition. In one or more embodiments, thepressure or force applied to the rectal wall is controlled and keptuniform across an ultrasound sweep. For example, referring back to FIGS.2A-C, prostate deformation in FIG. 2C is minimized throughout anultrasound sweep and consistent between positions of the ultrasoundprobe when compared to FIGS. 2A-B. Traditionally in robotics,end-effector force is measured and controlled using a force or pressuresensor mounted on the end-effector or the attached tool. However, thissolution may not be practical for TRUS procedures due to limitations inavailable acoustically translucent force sensors, sensor size and costimplications. In this embodiment, image constraints are used as asubstitute for a force or pressure sensor. For example, image derivedfeatures can be used to determine the pressure or force applied to therectal wall, such as image correlation, image strain correlation and/orusing an image confidence map based on comparison or differences betweendata for sequential scans.

In an embodiment, the pressure or force applied to the rectal wall iscontrolled and kept uniform by measuring the angular displacementbetween two consecutive ultrasound images to determine imagecorrelation. For example, when the angular displacement between twoconsecutive positions is small, then the captured image are highlycorrelative. Therefore, if the pressure or force applied to the rectalwall changes between consecutive images, then the consecutively capturedimages will not be as correlated. By monitoring changes in correlation(or de-correlation) between consecutively captured images, the positionof the ultrasound probe can be adjusted during the ultrasound sweep tomaintain consistent pressure on the rectal wall.

In another embodiment, the pressure or force applied to the rectal wallis controlled and kept uniform by measuring image strain. For example,strain images use ultrasound shear waves and/or acoustic radiation forceimaging to provide an estimate of strain on the tissue. Correlation inimage strain between two consecutively captured images should be similar(e.g., between images captured before and after a small change in probeposition). Metrics, such as cosine similarity, mutual information orother information theory metrics independent of image magnitude, areused to determine correlation between strain images. By monitoringchanges in correlation (or de-correlation) between consecutivelycaptured images, the position of the ultrasound probe can be adjustedduring the ultrasound sweep to maintain consistent pressure on therectal wall.

In yet another embodiment, the pressure or force applied to the rectalwall is controlled and kept uniform using an image confidence map. Forexample, an image confidence map is a learning model based on aprobability map that labels each pixel in the image with a value betweenzero and one (e.g., 0 to 1). The learning model is used to identifyacoustic shadows in an ultrasound image and/or an impedance mismatchbetween the ultrasound image and scanned surface. For example, acousticshadows at the far edge of the image (e.g., the edge opposite theultrasound probe) indicate that contact pressure is insufficient. Thus,when acoustic shadows in a confidence probability map are indicative ofinsufficient applied pressure, the position of the ultrasound probe canbe adjusted during the ultrasound sweep to maintain adequate pressure onthe rectal wall.

In an embodiment, the pressure or force applied to the rectal wall ismeasured using image correlation, image strain correlation and/or usingan image confidence map, either independently or in combination. Appliedpressure or force is computed on a patch, or portion, of an ultrasoundimage. For example, a portion of the image is computed in any pixel size(e.g., a portion that is w pixels by w pixels), and each patch returns ascalar value. Scalar values from N-patches are concatenated to generatea ND feature vector. Mapping between the tangent space of the ND featurevector and the Cartesian space is provided using the image JacobianJ_(I). Using the image Jacobian, the desired constraint to beimplemented is a small rate of change of the ND feature vector, writtenas equation 8:|J _(I) J _(k) Δq| _(n)≤ϵ_(I)  (Eq. 8)where n=1, 2. For example, an L1 norm squared computes each patchindependently and does not allow any patch to deviate beyond thespecified tolerance. In another example, an L2 norm squared treats eachpatch with equal weight. The feature vector is be scaled by a diagonalmatrix W, with diagonal elements representing the relative importance ofeach patch, as expressed in equation 9:|WJ _(I) J _(k) Δq| _(n)≤ϵ_(I)  (Eq. 9)

FIG. 4 illustrates a flowchart of an embodiment of a method forultrasound guided prostate needle biopsies using a robotic arm. Themethod is implemented by the system of FIG. 1 and/or a different system.A processor, such as a processor of an imaging system, performs acts401, 403, 405, 409, 411, and 413. The imaging system, using a probeguided by the processor, performs act 407. A user may assist or performacts 405 and/or 413.

Additional, different or fewer acts may be provided. For example, acts409-413 are not performed. As another example, act 409 is not provided.In yet another example, acts 401-409 are not provided. In otherexamples, acts for configuring for imaging, calibrating, activatingand/or imaging are provided.

The method is provided in the order shown. Other orders may be provided,and acts may be repeated.

At act 401, a three-dimensional image of a prostate is received, such asa preoperative MRI image. The three-dimensional image is received from athree-dimensional imaging system prior to the ultrasound procedure.Alternatively, a three-dimensional image of the prostate may be capturedinitially by scanning the patient in another act.

The data is received by scanning. Alternatively, the data is received byloading from memory or transfer over a network.

At act 403, a three-dimensional model of the prostate is generated usingthe three-dimensional image. For example, during preoperative planning,an automatic segmentation is performed on the three-dimensional imagedata (e.g., MRI map), and a patient specific anatomical model ofprostate is generated. Further, the biomechanical properties of theprostate may be determined (i.e., Young's modulus and the Poisson'sratio) using nominal values reported from literature, or thepersonalized values extracted from elastography scans of the patient areused. Any modeling may be used. The model is a three-dimensionalrepresentation of the prostate with or without surrounding tissues.Rather than a purely voxel intensity representation, other information,such as tissue properties, bio-mechanical properties or relationships,scale, orientation, or other information, may be included in the model.

At act 405, the ultrasound probe is guided into a position for athree-dimensional ultrasound sweep of the prostate. For example, theuser selects an operating mode for the robotic arm allowing the user tomove the ultrasound probe into position for the sweep. In oneembodiment, the user selects a freestyle mode that allows the user tomove the probe freely, only restricted by the mechanical limits of therobotic arm. Alternatively, or in addition to the freestyle mode, acompliant mode may be selected by the user to allow weightless movementof the ultrasound probe.

At act 407, a three-dimensional ultrasound sweep of the prostate isperformed. For example, when the user has localized the prostate, theuser manually or automatically acquires a three-dimensional sweep of theprostate. For example, the user selects an operating mode for therobotic arm allowing the user to manually perform the three-dimensionalsweep, or enables the system to automatically perform the prostatesweep. In one embodiment, an RCM mode is selected by the user torestrict the motion of the ultrasound probe about a fixed point on theultrasound probe, such as a distal end or tip of the ultrasound probe.The user then controls movement of the probe as constrained in the RCMmode. Alternatively, an “auto sweep” mode is selected to automaticallyguide the ultrasound probe through the three-dimensional ultrasoundsweep of the prostate. The system tracks the ultrasound images to rotatethe probe automatically based on prostate image information. In anotherexample, the system tracks the ultrasound images in RCM mode and/or“auto sweep” mode to maintain consistent and adequate pressure by theultrasound probe.

At act 409, the three-dimensional model of the prostate is updated usingimage data from the three-dimensional ultrasound prostate sweep. Forexample, the three-dimensional ultrasound image is automaticallysegmented and an updated three-dimensional model of the prostate isgenerated based on the ultrasound image. For example, thethree-dimensional model is updated to account for prostate deformationcaused by the ultrasound probe. Further, feature locations identifiedusing the preoperative MRI are updated in the updated three-dimensionalmodel, accounting for the effects of the prostate deformation byupdating feature locations in the three-dimensional model. For example,FIGS. 3A-B depict examples of a prostate segmented in an MRI image andan ultrasound image. FIG. 3A depicts a preoperative MRI image with aprostate outline 301 identified from the MRI image and a prostateoutline 303 from a corresponding ultrasound image (i.e., FIG. 3B). FIG.3B depicts a real-time ultrasound image with the prostate outline 301identified from the preoperative MRI image (i.e., FIG. 3A) and aprostate outline 303 from the ultrasound image. As such, changes to thethree-dimensional model are made to account for the change in shape ofthe prostate during the ultrasound procedure. In an example,registration between the preoperative MRI data and the real-timeultrasound data is established using the anatomical probe orientation.For example, locations from the three-dimensional preoperative ordiagnostic image are mapped with corresponding locations from thethree-dimensional ultrasound image. The probe orientation and positionis determined, and in combination with the ultrasound scan data, is usedto determine the orientation, position, and scale of the ultrasound datarelative to the MR data. Further refinement, such as warping, may beused. The updated three-dimensional model is generated usingpreoperative MRI data and the real-time ultrasound data to determineknown feature boundaries and may be enhanced using the nominal orpersonalized biomechanical properties of the patient.

The three-dimensional image data is fused and provided to the userreal-time. An image is generated from both data sets or separate imagesare generated from the separate data sets. The images show the sameanatomy form a same perspective, providing information from both typesof imaging. The ultrasound information provides representation inreal-time or during the procedure. The MR information may providegreater detail and/or additional information.

At act 411, a three-dimensional location in the prostate is identifiedas a target for collecting a biopsy core. For example, the location isidentified based on the updated three-dimensional model of the prostate.The location and size are determined using the fused information.

At act 413, a needle guide on the ultrasound probe is guided into aposition to collect the biopsy core, based on the identifiedthree-dimensional location in the prostate. For example, the userselects an operating mode for the robotic arm. For example, the userselects the RCM mode, restricting movement of the ultrasound probe aboutthe tip of the probe. The user then manually moves the ultrasound probeinto position to collect the biopsy core, passively guided by therobotic arm. Thus, based on the three-dimensional model, the user alignsthe probe so the biopsy needle guide is in position to target the biopsycore identified in the three-dimensional model. In another example, theuser selects an automatic mode where in the robotic arm automaticallymoves the ultrasound probe into position to collect the biopsy core. Inthis example, the user marks a target location in the three-dimensionalmodel, and the robotic arm is automatically moved (e.g., following RCMprinciples) to align the needle guide to the location in the prostatecorresponding to the marked location in the three-dimensional model.

When the needle guide is aligned with the desired biopsy core location,the user selects the lock mode. The lock mode maintains the position ofultrasound probe with respect to the target location, and restricts allmovement of the ultrasound probe. Acts 411 and 413 are repeated to takemultiple biopsy core samples.

Additionally, in one or more embodiments, a calibration act isperformed. For example, the robotic arm is calibrated for the handle andtip of the ultrasound probe with respect to coordinate system for thejoints of the robotic arm. Further, the system is calibrated for otherobject coordinates, such as the ultrasound machine itself, the patienttable, etc., to be used for collision avoidance constraints. Othercalibration may be used to align the preoperative data coordinate systemwith the robotic or ultrasound coordinate system.

Various improvements described herein may be used together orseparately. Although illustrative embodiments of the present inventionhave been described herein with reference to the accompanying drawings,it is to be understood that the invention is not limited to thoseprecise embodiments, and that various other changes and modificationsmay be affected therein by one skilled in the art without departing fromthe scope or spirit of the invention.

We claim:
 1. A system for ultrasound guided prostate needle biopsiesusing a robotic arm, the system comprising: a processor configured toreceive a preoperative three-dimensional image of a prostate, whereinthe processor is further configured to generate a three-dimensionalmodel of the prostate based on the received image; a multi-link roboticarm comprising: a plurality of joints; an end-effector attached to oneof the plurality of joints; an ultrasound probe attached to theend-effector, wherein the ultrasound probe is operably connected to theprocessor to capture an ultrasound image, wherein the preoperativethree-dimensional image was captured using a different imaging modalityfrom the ultrasound image, wherein the processor updates thethree-dimensional model of the prostate based on the captured ultrasoundimage; a position sensor for at least one of the plurality of joints,the position sensor operably connected to the processor to map theposition of the ultrasound probe with the updated three-dimensionalmodel; and a force sensor operably connected to the processor, whereinthe force sensor is configured to detect input forces to theend-effector, wherein the processor, the position sensor and the forcesensor comprise a control loop configured to control manipulation of theultrasound probe based on updates to the three-dimensional model of theprostate, and wherein the robotic arm is configured to operate in afirst operating mode restricting movement of the ultrasound probe abouta distal end of the ultrasound probe to maintain pressure on theprostate based on consecutively captured ultrasound images in automaticperformance of a three-dimensional ultrasound sweep of the prostate. 2.The system of claim 1 wherein the force sensor is attached to a handleof the ultrasound probe.
 3. The system of claim 2 wherein the roboticarm is configured to operate in a plurality of other operating modes torestrict movement of the ultrasound probe, the plurality of otheroperating modes comprising: a second operating mode restricting movementof the ultrasound probe by mechanical limits of the robotic arm; a thirdoperating mode passively guiding movement of the ultrasound probe into aposition to collect a biopsy core; a fourth operating mode restrictingall movement of the ultrasound probe; and a fifth operating moderestricting movement of the ultrasound probe based on analysis ofcaptured ultrasound images.
 4. The system of claim 3 wherein theprocessor is configured to identify the biopsy core based on thereceived three-dimensional image of the prostate and the capturedultrasound image of the prostate.
 5. The system of claim 1 wherein thepressure is maintained on the prostate based on de-correlation betweenthe consecutively captured ultrasound images.
 6. The system of claim 1wherein the pressure is maintained on the prostate based on a differencein a strain on prostate tissue between the consecutively capturedultrasound images.
 7. The system of claim 1 wherein the pressure ismaintained on the prostate based on an image confidence map generatedbased on the consecutively captured ultrasound images.
 8. The system ofclaim 1 wherein the pressure is maintained on the prostate basedde-correlation between the consecutively captured ultrasound images,strain on prostate tissue between the consecutively captured ultrasoundimages and an image confidence map generated based on the consecutivelycaptured ultrasound images.
 9. The system of claim 1, wherein thecontrol loop is configured to provide adjustable compliance of themulti-link robotic arm.
 10. The system of claim 1 wherein the controlloop is configured to restrict motion of the ultrasound probe.
 11. Thesystem of claim 1 wherein the preoperative three-dimensional modelcomprises a voxel representation of the prostate with prostateinformation.
 12. The system of claim 11 wherein the prostate informationcomprises tissue properties, biomechanical properties, or biomechanicalrelationships.
 13. The system of claim 1 wherein the processor isconfigured to identify a size of a biopsy core based on the receivedthree-dimensional image of the prostate and the captured ultrasoundimage of the prostate.